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System and method for image enhancement. 

(57) Analog video data Input is received by a video 
capture system and digitised. From the pels of 
the captured Image, a gray scale level histog- 
ram is constructed with the system processor, 
and normalised to established white-black re- 
ferences similar to the contrast control function 
at image capture time. Peaks likely attributable 
to image artifacts are limited. The image pro- 
cessor then performs a second order best fit 
regression of the histogram from which the 
image is thereafter conrected. An automatic 
correction of the image contrast Is thereby 
effected, wherein capture artifacts are reduced 
while retaining those of the image. In an alter- 
nate embodiment, instead of limiting peaks, an 
iterative best fit is performed, wherein the peaks 
are weighted In a subsequent best fit pass as a 
function of their distances from the curve fit 
from a prior best fit pass. 
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This invention relates to a system and method of 
automatic image enhancement. 

In the field of digital video technology, a problem 
often arises with captured images similar to that 
encountered in the photographic arts, namely the con- 
trast of the image is defective and in need of correc- 
tion. 

Several methods have been employed in the past 
to correct such defects. For example, it is known in the 
satellite image processing art to effectively flatten the 
gray scale histogram of the image without attempting 
to sort out and treat differently the artifacts attributable 
to the image, which are desirable; from those of the 
image capture process which are undesirable 
anomalies. 

Such technology provides a high entropy image 
and is quite helpful with respect to images encoun- 
tered in space exploration. However, when applied to 
natural images more typically encountered in mul- 
timedia work, the result can often include very disturb- 
ing and unnatural appearing artifacts associated with 
the capture process as, for example, in a grainy 
appearance of the sky in an outdoor image. 

Technology has further been developed to permit 
manual adjustment of contrast levels as, for example, 
in equipment employed in photographic film proces- 
sing to produce photographic images. A correction 
curve employed by the equipment may be adjusted by 
hand, for example, in a commercial slide service 
establishment, wherein an operator may raise high- 
lights, lower shadows, raise medium grays, lower mid- 
dle grays, or the like, so as to enhance or reduce 
undesirable characteristics of the photograph such as 
saturation. 

In a related area, it has been known to provide 
various forms of video signal processing equipment 
which permits the operator to manually adjust the con- 
trast or gamma of the captured video signal. White 
such manual techniques provide the opportunity for 
great flexibility and artistic input these same charac- 
teristics also give rise to numerous problems. 
Perhaps the most obvious of these is the need for skil- 
led operators and their subjective judgement, as well 
as the repeatability and consistency of the artistic 
judgements which are being made. 

It was accordingly highly desirable in some appli- 
cations, such as in commercial multimedia systems, 
to make provision for some form of automatic gamma 
or contrast adjustments in the video image whereby 
these artistic functions might be performed heuristi- 
catly without user intervention. Certainly various auto- 
matic contrast schemes have been devised in the 
image processing and photographic arts in seeking to 
remove the human variable, as. for example, in the 
automatic range setting of black-white levels or auto- 
matic brightness controls in some fonms of video 
equipment. In some cases, for example in commercial 
photography, an average of gray scales in the image 



is sometimes taken and the brightness level set as a 
function of this average. In more sophisticated equip- 
ment, automatic processing circuitry may examine the 
image for the brightest and darkest levels and set the 

5 contrast as a function of these levels. 

A serious problem with these automated 
approaches, however, is that again there is no atten- 
tion given to seeking to discriminate between image 
artifacts and undesirable capture artifacts. A tech- 

10 ntque was thus highly desired which could automati- 
cally perform basic contrast correction on captured 
images without user intervention while at the same 
time removing from the image only the artifacts intro- 
duced by the capture process. 

15 The invention provides a method for automatic 

image enhancement of an image, comprising 
generating a histogram from video data representing 
said image; generating a best fit regression function 
of said histogram; and modifying said video data as a 

20 function of said regression function. 

The invention further provides a system for auto- 
matic image enhancement comprising means for 
generating a histogram from video data representing 
said image, means for generating a best bit regres- 

25 sion function of said histogram; and means for mod- 
ifying said video data as a function of said regression 
function. 

In order that the invention may be understood, 
preferred embodiment thereof wilt now be described 
30 by way of example with reference to the accompany- 
ing drawings, in which: 

Fig. 1 is a functional block diagram depicting a 
computerised video image capture and processing 
system. 

35 Fig. 2 is a more detailed block diagram of the sys- 

tem of Fig. 1. 

Fig. 3 is a schematic representation of the pro- 
cess for capturing an image and effecting automatic 
gamma correction to generate a corrected digital 
40 image in accordance with the invention. 

Fig. 4 is a more detailed flow diagram of the step 
of establishing a second order best fit depicted in Fig. 
3. 

Fig. 5 is a schematic illustration of a captured 
45 image illustrating the cropping of the edge data. 

Fig. 6 is a simplified illustration of a histogram of 
gray scale levels derived from the captured image of 
Fig. 5. 

Fig. 7 is another illustration of the histogram of 
50 gray scale levels of Fig. 6 indicating establishment of 
white-black references and the calculated white-black 
range on the histogram employed to gather statistics 
in accordance with the invention. 

Fig. 8 is another histogram of gray scale levels 
55 derived from that of Fig. 7 wherein normalisation and 
sample limiting has been effected, whereby at the 
high end excess contrast giving rise to noise and 
single high points giving rise to uncorrectable satu- 
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ration are avoided and further whereby low ends are 
limited to ensure a positive best fit curve. 

Fig. 9 is an illustrative second order best fit curve 
of the data depicted in Fig. 8. 

Fig. 10 is a pair of delta arrays derived from the 5 
data of Fig. 9. used to correct the image of Fig. 5. 

Fig. 11 depicts the histogram of Fig. 6 in an inte- 
grated transfer curve more common to those familiar 
with film-based images. 

First, a description of the problem addressed will io 
be provided along with a more general description of 
the system for effecting automatic gamma correction. 
This will then be followed by a description of a video 
capture and processing system with reference to Figs. 
1 and 2 adapted for use In effecting the invention. 15 
Next, a high level description of the digital processing 
of the captured image will be given with reference to 
Figs. 3 and 4. This will then be followed by a more 
detailed description of a particular embodiment of the 
digital signal processing with reference to Figs. 5-10, 20 

The system to be described provides the artistic 
function heuristically without user intervention by 
automatically performing basic contrast correction on 
a captured image. In order to do so, as a broad over- 
view of the functions to be performed (which will be 25 
disclosed later in more detail), the system operates to 
analyse the image, determine limitations in the cap- 
ture process, separate anomalies caused by capture 
system limitations from desirable features of the 
image, and thence remove artifacts introduced by the 30 
capture process. Although a particular embodiment 
disclosed is concerned with operating only on the 
shape of the gray scale, (thereby effecting contrast) 
colour and sharpness will simultaneously be 
improved subjectively as a consequence. 35 

In order to more cleariy understand the problem 
addressed by the present invention, it will be 
appreciated that if a test image containing ail levels of 
gray uniformly distributed therein was captured with a 
system having capture anomalies, a histogram built 40 
from the capture of this test image could be used 
directly in correction routines for subsequent more 
useful images to remove these capture anomalies. 

One problem in the real worid, however, is that 
such a test image often is simply not available. As but 45 
one example, an image in a book we desire to digitise 
has already been captured by the camera which 
includes the capture anomalies, along with the infor- 
mation we want to preserve defining the image also 
added in. In the simplest terms, the system disclosed so 
herein seeks to see through the characteristics of the 
image to the underiying defects of the capture pro- 
cess. It then seeks to essentially regenerate the test 
histogram we would have obtained if a plain gray 
scale test image had first been captured from which 55 
to calibrate. 

Before further discussing broadly the technique 
implemented herein, certain assumptions must be 



noted about the capture process itself. It is first 
assumed that peaks in the gray scale histogram of an 
image imply areas of relatively solid colours indicating 
an image artifact and not a capture artifact. These 
peaks will accordingly be ignored as described in the 
limiting step of the preferred embodiment. 

A second assumption relates to the degree of 
smoothness desired in the curve fitting to the histog- 
ram which in the preferred embodiment is effected by 
means of a second order or quadratic fit. The higher 
the order, i.e. the larger the n in X" employed in the 
curve fit, the more detailed variations of the actual his- 
togram curve are being accounted for in the curve fit- 
ting. Higher order curve fitting (employing terms such 
as X3 and X^) are empirically found not to map to defi- 
nite physical capture defects. This is contrasted with 
X and X2 terms which are found empirically to map 
quite closely to gamma and saturation in the types of 
images normally encountered with respect to conven- 
tional multimedia work. By accounting for these faster 
changes in the histogram curve by higher order curve 
fitting, the danger is increased that image artifacts are 
being picked up. Consequently it is a feature of the 
invention that second order fitting has been applied so 
as to ignore these artifacts. In other words by perform- 
ing a second order best fit. this effectively serves as 
a way of seeing through a class of image anomalies 
as we otherwise would have done if the aforemen- 
tioned test gray scale was available. 

It is thus assumed, as one feature of the inven- 
tion, that the capture process warps the gray scale 
curve with a quadratic of the form A + Bx + Cx2. Any 
higher order terms, i.e. any more abrupt changes, 
arise from image features we desire to preserve or 
from uncorrectable capture defects such as saturated 
whites and blacks as previously noted. The "A" term 
simply relates to the number of pels in the image and 
is algorithmically uninteresting. The remaining terms, 
however, represent the two major factors which can 
go wrong in image capture which are contrast and 
saturation. 

The "B" empirically measures or maps to gamma, 
i.e. true contrast. If this "B" coefficient is negative, this 
implies there are too many dark pels and not enough 
light pels and thus the image appears dark and "con- 
trasty". A negative B quantifies a very common prob- 
lem when videographically capturing from a print or 
slide, for example, in which the gamma of both the 
photographic media and the electronic media add 
together. A positive B quantifies an image with too lit- 
tle true contrast. The image may still have full blacks 
and whites, but in between the image is too light, high- 
light detail is faint, and the image looks lifeless. Stated 
differently a negative B image typically corresponds to 
a high gamma containing tots of dark values, referred 
to as "low key" in the art, while a positive B image 
would be low gamma and contain lots of white values, 
referred to as "high key". 
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The "C" term quantifies saturation, seen as the 
"toe" toward the intersection of the axis in the gray 
scale curve toward black, and the "shoulder" of the 
curve at the rightmost end thereof. A positive "C" typi- 
cally indicates that either or both the highlights or 
shadows are saturating, indicated by a rise at the end 
of the gray scale graph. The "B" term works with the 
"C" term to specify if the shadows, highlights, or both 
are being compressed by saturation. It is for this 
reason that the B term needs to be interpreted in light 
of the C term value In accordance with the invention. 
Sometimes an image produces a negative C, it will be 
noted, as could happen if there were brilliant high- 
lights which should not be clipped but rather need to 
be compressed to allow gray scale room for the rest 
of the image to be seen. 

Broadly, the process of the invention is comprised 
of three steps. First the image is digitised or captured 
with an appropriate digital video capture system such 
as that hereinafter described generally. With the thus- 
captured pets in storage, a gray scale histogram is 
created. The creation of histogram in this first step 
essentially builds a graph showing the frequency of 
occurrence for each level in the gray scale. A schema- 
tic of such a representative histogram may be seen in 
Fig. 6. This graph will contain all Information about the 
image employed in the Image processing steps util- 
ised to generate the corrected image. As an example. 
If the original image included too many dark shadows, 
a hump would appear at the bottom of the graph in the 
vicinity of reference numeral 84. whereas if the origi- 
nal image was underexposed, the top of the graph 
would approach 0 as indicated as reference numeral 
85 in Fig. 6. 

As a next step in the Image processing, the Image 
is thence corrected by redistributing gray levels so the 
gray scale graph is more uniform. For example, if the 
original had too many dark shadows, each gray scale 
step in the shadows of the original might translate to 
two gray scale steps in the shadows of the corrected 
image, thereby spreading steps over a wider range 
and reducing the hump at the bottom of the graph. In 
like manner, if the original was underexposed, the top 
of the gray scale would be compressed during correc- 
tion, compressing rarified gray scale steps into former 
steps, thus raising the number of pels at each step up 
to a norm. By making each state in the gray scale 
more nearly equal in probability across the image, the 
"information content" of the image goes up In accord- 
ance with formulas from Information theory welt 
known in the art. Because the human eye desires to 
perceive "information" presented clearly, the aesthe- 
tic appeal of the Image will thereby usually be 
improved. 

Thus, it will be appreciated that in the second step 
of the process, stated differently, the process is seek- 
ing to synthesise a gray scale histogram, e.g. to 
recover what would have been obtained as a histog- 



ram from the test gray scale disposed In front of the 
camera. In regressing to such a syntheslsed gray 
scale histogram. It should be readily apparent that this 
regression from the histogram of the capture is seek- 

5 ing to predict the histogram of the ideal test gray 
scale, In other words, what the gray scale histogram 
would have looked like based upon a priority 
knowledge of what artifacts would belong to the image 
and which artifacts would belong to defects In the cap- 

10 ture process. Thus the process seeks to sort out these 
differing image and capture artifacts in performing this 
regression. 

An extreme correction would rearrange the gray 
scale so that each gray step in the corrected image 

15 had the same number of image pixels associated with 
it. Such an extreme correction would. In a mathemati- 
cal sense, yield the most "Information" since each 
state would now have equal probability. However, the 
gray scale curve could become disjointed causing 

20 small area artifacts. Moreover, large areas of the 
Image with the same gray level such as the aforemen- 
tioned sky In the prior art associated with space exp- 
loration image processing, would be assigned too 
much contrast and would therefore appear grainy. 

25 Accordingly, a difficult aspect of the processing of the 
invention is in the determination of what character- 
istics of the gray scale graph are in fact from these 
capture artifacts which should be corrected and which 
parts are from capture artifacts which cannot be cor- 

30 rected, and further which parts are from image 
characteristics that should not be corrected. As an 
example of portions which cannot be corrected, high- 
lights may have been so overexposed and saturated 
as to essentially have disappeared. 

35 As a third step in the image processing in accord- 

ance with the invention, from aforementioned regres- 
sion to the gray scale histogram which the process 
seeks to synthesise and from the captured Image, the 
Image Is thereby corrected according to the best fit. 

40 Relative to effecting this best fit, one of the most dif- 
ficult problems addressed by the subject Invention Is 
in the determination of which artifacts of the curve are 
caused by the image which are desired to be preser- 
ved versus those caused by defects in the capture 

45 process itself. Examples of the latter include under- 
exposing film, excessive gamma, or some portion of 
the image which has saturated. Certain character- 
istics of the image have been discovered to Indicate 
which sort of artifact is present and are employed in 

50 the image processing in an effort to attempt to sepa- 
rate them in the best fit. 

Now that a general overview of the Image proces- 
sing of the invention has been presented, a represen- 
tative digital video capture and processing system 

55 adapted to the present invention will be described with 
reference to Figs. 1 and 2 after which will follow a 
more detailed description of a particular preferred 
embodiment of the processing. One such system 
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which may be employed in the manner depicted in 
Figs. 1 and 2 is the IBM Audio Visual Connection 
(AVC) system which includes both the hardware and 
software necessary for the capture, processing, and 
display of analog video data. This system is manufac- 5 
tured by the IBM Corporation of Armonk, New York. 
Supporting documentation for such a system may be 
found in the following publications which are herein 
incorporated by reference: IBM Audio Visual Connec- 
tion User's Guide and the Video Capture Adapter Ins- io 
tallation and Technical Reference Manual , both of 
which are available from the IBM Corporation.' 

Fig. 1 depicts in block diagram form a digital com- 
puter system 1 0, such as the AVC system. System 1 0 
may take the fonn of a digital personal computer and i5 
includes a keyboard 6 for providing user input which 
is coupled to a central processing unit or host PC 4 in 
a conventional manner. Included in the host 4 are 
memory devices, which may comprise both volatile 
and non-volatile memory storage devices and which 20 
may include auxiliary memory devices such as mag- 
netic disk or tape storage. 

Host 4 is also coupled to one or more display driv- 
ers (not shown) which are utilised, in a conventional 
manner, to provide a visual display of video images by 25 
coupling appropriate control and data signals from the 
host 4 to the display 5 and one or more monitors 7. It 
will be appreciated that the display 5 and monitor 7 
may be provided by a variety of devices such as those 
adapted to receive analog video in varying formats 30 
such as RGB, Y/C, NTSC, and the like. 

In the system 10 shown in Fig. 1, a video camera 
3 is further included having an analog video link 32 to 
the host 4. The camera 3 is shown to be trained on an 
object 2 having visual images to be digitised by the 35 
system 10. The camera 3 transfers the visual image 
of object 2 into analog electronic signals carried on the 
link 32 to the host 4 wherein they will subsequently be 
digitised, processed and reconverted into analog form 
for output on analog output link 34 after which they are 40 
displayed on one or more monitors 7. The system 10 
may, of course, include other components. The gen- 
eral digital computer and display portions of the sys- 
tem 10 may be be provided for example by means of 
an IBM Model 80 PS/2 computer (IBM and PS/2 are 45 
trade marks of InternationalBusiness Machines Cor- 
poration of Armonk, New York.) 

The host 4, as is common in personal computers, 
includes a bus system facilitating data, control, and 
address I/O to various components of the system in a so 
manner well known in the art. In particular with respect 
to the aforementioned Model 80 PS/2 computer, an 
IBM Micro Channel bus system is preferably 
employed. (Micro Channel is a trademark of the Inter- 
national Business Machines Corporation.) However, 55 
the invention is not intended to be so limited to a par- 
ticular bus staicture and contemplates use with other 
bus structures. More detailed infonmation regarding 



this microchannel bus structure and operation may be 
obtained with reference to the technical manuals 
accompanying the Model 80 PS/2 machine. 

Although analog video input signals on link 32 
have been shown as originating from the camera 3, it 
will be readily appreciated that such analog video 
source material may be generated from a number of 
other sources such as the familiar VCR, VTR (video 
tape recorder), or the like. These sources of analog 
video signals may be in a number of formats such as 
the aforementioned RGB, Y/C, composite video. 
Although only one source of such formatted video 
information is shown in Fig. 1 , a plurality of multiformat 
sources may be accommodated for subsequent digi- 
tising in the host 4 of their respective video images or 
data, processed and subsequently re-converted back 
to analog video signals for simultaneous display on 
one or more display devices. A computer monitor 5 
has been shown in Fig. 1 which conventionally com- 
prises a portion of a general personal computer sys- 
tem including the host 4 and is used for conventional 
purposes of I/O with the host 4. However, as herein- 
before noted, a monitor 7 has further been included 
for display of these re-converted analog video signals. 

Referring again briefly to the bus structures 
associated with personal computer systems 10, it is 
conventional to provide in the various available hosts 
4 a plurality of adapter card or expansion slots. The 
purpose of these slots is to enable a user to custom 
configure a system to perform desired functions. An 
adapter card in one of the slots may carry connectors 
for receiving raw analog video source data in multiple 
formats and multiplexing/control hardware and the 
like on the card which, under computer control by 
interfacing with the aforementioned bus, will select 
one of these signals for digitisation. This adapter card 
may further include the analog-to-digital circuitry 
necessary to digitise this preselected signal. By inter- 
connection of the adapter card with the bus, proces- 
sing and editing of the thereby digitised image under 
control of the host 4 computer through the bus to elimi- 
nate motion artifacts is provided. A further function of 
the adapter card is to then re-convert the processed- 
/edited digitised video information into analog form, 
such function also being performed on board on the 
card by provision of appropriate digital-to-analog con- 
version circuitry. 

Finally, encoder circuitry is provided on the adap- 
ter card in order to convert the digitised video signals 
into multiple analog fonnats. Multiple connectors are 
provided on the card each carrying a different one of 
these re-converted digitised signals in a different one 
of the conventional analog video formats. In this man- 
ner, by connecting a monitor to each such connector 
adapted to receive that connector's particular signal 
format, simultaneous display of the video image in 
multiple fonmats results. It will be noted that this re- 
conversion is also under computer4 control inasmuch 
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as the adapter card and associated encoder circuitry 
is plugged into the interface bus of the computer 4. 

With reference now to Fig. 2, a more detailed 
functional block diagram of the system 10 of Fig. 1 is 
illustrated which will now be described. 

As shown at line 32 analog video input signals in 
a variety of formats are received by the video adapter 
1 6 of the invention. The video adapter 1 6 is connected 
to the host system 12 by the video device driver 18 
which receives status from the adapter on tines 36 
and sends commands and data to registers of adapter 
16 on line 38. The bus of the system 10 is illustrated 
graphically in part by output on line 26 going from 
adapter 16 into the video buffers 28 inside the host 
processor 12. Further function of this bus is illustrated 
by line 30 indicating information stored in this video 
buffer 28 may be loaded into host storage such as a 
PC hard disk 22. From storage 22 via the bus, again 
illustrated by a line, 24, data can be received from the 
disk and put back into the buffer 28 of host 12. From 
these buffers 28 data may also be read and then sent 
back via the host bus connection to the adapter 16 on 
line 32 to load video data into the video memory of the 
adapter 16. 

A video device driver 18 has a video application 
programming interface 20 which directly interfaces as 
shown by line 15 with the video capture and edit 
capability or function 14 of the host system 10. The 
application programming interface 20 provides micro- 
commands for the video capture and edit process that 
are ultimately delivered to the video adapter 16 via the 
device driver box 18. As a result of execution of the 
video capture and edit program 14 commands are 
sent via lines 40 to and status information is received 
via lines 42 from the system processor of host 12. This 
results in an image being displayed on the screen of 
one or more monitors 7, Fig. 1. The screen 17 of host 
1 2 will generally be reserved for display of on-source 
material, i.e., the more conventional I/O data 
associated with personal computer systems. Such 
data includes operator input from keyboard 6, and 
user- interface information from the host 1 2 for operat- 
ing the capture and display features of the invention. 
The displayed image may be live or a composite 
image including images controlled by the function 
selected by the video capture and edit module. The 
path 15 between the video capture and edit function 
1 4 and the application programming interface 20 is via 
microcommands which are then in turn converted into 
individual separate commands and status requests 
across lines 36 and 38. As hereinbefore suggested 
functions indicated by lines 32, 30, 24 and 26 are all 
actually handled via the bus itself of system 10 and 
generally represent disk 22 I/O and adapter card 16 
I/O. 

It will be appreciated from the foregoing that once 
the image has been captured in the form of digitised 
representations of pets of various frames of video 



data by means of the system depicted in Figs. 1 and 
2, software may be provided for the host processor 12 
for manipulating such data in any desired manner. It 
is of course a feature of the invention to provide such 

5 a computerised process for operating on this digitised 
data in the manner herein described in accordance 
with the invention to effect the desired image correc- 
tion. Thus an image processing application program 
21 of the invention may be provided to interface with 

10 the video application programming interface 20 so as 
to effect the desired manipulation and transformation 
of these digital representations of the pels. The pels 
are stored in the PC disk 22 and buffers 28 prior their 
restorage on PC disk 22 or being output through video 

15 adapter 16 as analog 34 for display of an enhanced 
image. The system 10, in accordance with the inven- 
tion, has thus generated an enhanced image eliminat- 
ing the undesirable capture artifacts by the 
processing software interacting with the system 10. 

20 Now that a description of the capture system has 

been provided having the capability of image capture, 
processing in accordance with invention under 
software control and display, the detailed description 
of the processing of these digital representations of 

25 capture pels by the processing application in accord- 
ance with the invention will be provided. 

In Fig. 3 there may be see a schematic represen- 
tation of the foregoing general description of the cap- 
ture and processing of the invention. The image 

30 captured by a suitable source such as a camera 50 is 
digitised as shown as block 52, whereupon the video 
system constructs the histogram at block 54 to be 
hereinafter detailed with greater specificity. The his- 
togram thus created thence has a best fit curve gen- 

35 erated to model it, shown at block 56. The original 
image digitised at block 52 is then functionally altered 
in accordance with the best fit curve generated at 
block 56 as shown schematically by the arrows 58 and 
block 60. resulting in a digital image corrected auto- 

40 matically for contrast. 

The best fit block 56 of Fig. 3 is further expanded 
in Fig. 4. First, the gray scale range is limited to an 
appropriate range in block 62. Within this range, ext- 
remely high or low histogram counts are limited In 

45 block 66, and the quadratic best fit of the resulting his- 
togram within the limited gray scale range is com- 
puted in block 68. The range and best fit date are 
integrated in block 70 to give the overall best fit, from 
which the gray scale correction function is found in 

50 block 72 that will be applied to correct the original 
image. 

In the detailed illustration of the processing of the 
captured image, such a digitised image may he seen 
schematically depicted as image 80 in Fig. 5. The pro- 
55 cessing to effect image correction of the invention 
may be implemented in the application program 21 of 
Fig. 2. This program may be readily implemented by 
one skilled in the programming arts with reference to 
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the accompanying description and figures. 

In the first detailed step of the processing, which 
may, for illustrative purposes be a 480 x 640 image 
system, the centre of the image of Fig. 5 would first be 
scaned as, for example, from X = 40 to 599 and Y = 5 
40 to 439, Doing so eliminates potential problems 
associated with image edges well known in the art. As 
one example, by international standard, a conven- 
tional 35 millimetre negative film area is defined by 24 
x 36 millimetres. The outer portion of this area is typi- io 
cally covered by the frame and thus in the processing 
herein described, these edges would be deleted from 
the image. As but another brief exam pie of edge prob- 
lems obviated in this step, captured images may fre- 
quently include too much undesired image is 
characteristics such as sky or extreme shadows. 

In this detailed example, it is assumed that the 
red, green, and blue records of image 5 are digitised 
to 6 bits and therefore can take on 2® = 64 possible 
values. These discrete values are represented as dis- 20 
Crete points along the horizontal axis in Figs. 6-9. 

The second detailed step in the image processing 
of the invention effects the previously described build- 
ing of a histogram of gray scale levels as shown in Fig. 
6 from the captured "cropped" image of Fig. 5. For 25 
each pel in the cropped image of fig. 5, a red, green, 
and blue level will be determined. The gray scale 
graph elements pointed to by the red, green, and blue 
levels will each be incremented by 4, 4, and 1 respect- 
ively. Before discussing the numerous rationales 30 
behind construction of a histogram in such a manner, 
several characteristics of an example the image ref- 
lected by the resultant histogram may be noted with 
reference to Fig. 6. These include the presence of 
"weak" blacks at reference numeral 82 wherein con- 35 
trast is so low that shadows ail appear substantially 
the same. Such a image exhibits further character- 
istics in need of correction such as blocked up 
shadows at 84, portions of the image at 86 which are 
too dark and "contrasty" and elements of the image 40 
evidencing uncorrectable saturation such as those 
shown as reference numeral 88. It will also be noted 
that depicted in the histogram of Fig. 6 are represen- 
tative image artifacts 90 which are desirably to be pre- 
served while at the same time eliminating those 45 
associated with a defective capture process. 

As to the rationale for construction of the histog- 
ram in such a manner, first in effecting a gray scale 
level, wherein all colours are lumped together, it has 
been empirically observed that brightly coloured so 
scenes are low key under one colour and high key 
under another colour. Correcting each colour image 
separately would in many applications result in decol- 
ourising many scenes. Still further it will be noted that 
the gray scale has been incremented by dissimilar 55 
values relative to the red. green, and blue levels. It 
has further been observed empirically that in some 
colour-mapped images, the blue record has higher 



noise and therefore blue is an unreliable indicator of 
saturation and has ascribed thereto a lower incre- 
menting value relative to red and green. Moreover, 
red, green, and blue are each counted separately 
rather than just counting luminance. In this manner, 
bright colours are correctly accounted for at both ends 
of the histogram even though luminance would count 
incorrectly in the middle of the histogram. It will be 
noted how monochrome images of colour prints seem 
to have more grays than images photographed origi- 
nally in black and white. 

Finally, although green may typically have more 
visual "weighting" and contain the least noise, the red 
image is frequently more important in many scenes, 
tends to have a higher contrast record, and is a better 
indicator of problems so in net. both green and red 
have been counted equally. Blue, on the other hand, 
frequently has the highest noise and the lowest visual 
weight, and images more randomly, and is accord- 
ingly ascribed little weight. 

Referring now to Fig. 7, this figure is intended to 
schematically illustrate the next step in the image pro- 
cessing, namely the establishment of white and black 
references. First, a total is derived representing the 
sum of the 64 gray levels, i.e. the total number of all 
colours counted which is equal to (total number of 
pels) X (4+4+1). Next a number "black 200" is deter- 
mined as the highest gray scale value number, such 
that the umber of all lower levels are less than or equal 
to total/200. In order to do this, the process will desi- 
rably count up from black zero and when the sum of 
all R. G, B is greater than totai/200. that level is thence 
assigned as the value of black 200 shown at reference 
numeral 92 in Fig. 7. 

Similariy, a black 32 value will be determined and 
ascribed at the value representing total/32, depicted 
schematically in Fig. 7 as reference numeral 94. Simi- 
lariy, values for white 32 and white 200 at the top of 
the histogram are determined shown as reference 
numeral 96. There are several rationales for the 
foregoing establishment of black and white refer- 
ences. Essentially this step is performing the same 
normalisation available with the contrast controls at 
image capture time. Usually, if an image was captured 
conrectly, the "200" points will be at the extreme 
values of 0 and 63 and the "32" points will be very 
close to these extreme points. The "200" cutoff points 
will be the new white-black levels, however statistics 
will be drawn fnam the narrower range so that satu- 
ration effects are more cleariy concentrated at the 
ends. The difference between the "32" and "200" 
points allows a smooth rolloff for specular reflections 
in live images wherein the correction for these specu- 
lar artifacts are not available at capture time. Exam- 
ples of specular reflections include those of chrome 
and sunlight. If such highlights are captured, one fre- 
quently has to darken the image to cover the full range 
from black to white such that a white wall for example 
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will appear undesirably darkened. Whereas specular 
reflections cannot be Ignored completely, in accord- 
ance with this step of the method they are not used to 
set the pure white value and are thus desirably rep- 
resented by a small range beyond the normal white 
level. In like manner, In a real worid lmage» a black 
cutoff point is usually not abrupt but rather darker 
values will tend to saturate into black, evidencing 
some vignetting caused by the camera and other cap- 
ture anomalies. In other words, a histogram would not 
be expected to rise to saturation and drop 
immediately to zero but rather will roll off as shown in 
Fig. 7. However, in the curve fitting in accordance with 
the invention, it is desirable for the process of the 
curve fitting not to see this rolloff but rather to see 
more uniform curve to fit to or otherwise an inapprop- 
riate best fit will result. It is for this reason that rolloff 
is eliminated in accordance with this step of setting 
white and black references. 

The next step in this detailed embodiment of the 
image processing is to normalise the histogram so the 
counts at all gray scale levels average one. This may 
be effected by multiplying all gray scale counts by 
(white 32 minus black 32 plus one) / (total count times 
63/64). It will be observed that this 63/64 term is actu- 
ally between 62/64 and 1 for any specific image. 
These use of white 32 and black 32 gray scale points 
normalises the values within the range 98 of Fig. 7. 
This is the range within which the best fit will be 
derived. 

Next, limiting of the curve is effected thereby limi- 
ting the extreme points of the histogram which are 
probably image artifacts such as those shown at refer- 
ence numeral 90 in Fig. 7 and 100 in Fig. 8. A com- 
parison of the ordinates of these image artifacts at 90 
and 100 and unconrectable saturation elements at 88 
and 102 of Figs. 7 and 8, respectively, will graphically 
illustrate this limiting function. In addition to the 
aforementioned limiting or pulling down of the ex- 
treme high points, the low points in a histogram in like 
manner will be pulled upas shown from a comparison 
of Figs. 7 and 8. There are several rationales for the 
foregoing limiting process. First, limiting at the high 
end is found necessary because by boosting contrast 
too high, this simply results in noise in the image. 
Also, it is desirable to ignore single points which are 
very high because this saturation is uncorrectable. 
Also, limiting at the tow end is provided to ensure that 
the best fit curve remains positive. A typical limiting 
function reassigns the count at each level of the gray 
scale as follows. If the count is less than 0.5. then the 
count is assigned 0.25 + count/2. But if the count is 
greater than 2, then count = 1 + count/2, and if the 
count is still greater than 3. then count = 3. 

The next step in the image processing is to effect 
the previously described best fit in the form A + Bx + 
Cx2. This second order best fit is perfonmed between 
the black 32 level and white 32 level shown at refer- 



ences numerals 94 and 96 of Fig. 9. with the resulting 
best fit curve being depicted therein as curve 104. 
This curve fitting may be effected in a number of man- 
ners well known in the mathematical arts with the 

5 assistance of computer processing provided by the 
system depicted in Figs. 1 and 2. 

Next in accordance with the invention a smooth 
gray scale graph is created, i.e. given the second 
order best fit curve 104 of Fig. 9, and the overall range 

10 of black 200 at 95 to white 200 at 97, the best fit curve 
is "smoothed" by providing an overall best fit over the 
entire range by tapering the edges off as shown for 
example at 106 in Fig. 9. 

In order to translate the curve of fig, 9 into a look- 

15 Up table to conveniently apply the confection of that 
curve to each pixel of image 5, first a "delta" array is 
created with elements at 0.5, 1.5, 2.5, ... 61.5 and 
62.5. Between block 32 plus 0.5 and white 3.2 minus 
0.5, the value of the function f(x) = A + Be + Cx^ with 

20 A, B, C determined in the previous step, is evaluated 
and the results are placed in this delta array 108. Next 
this f(x) is eveluated at black 32 and assigned a value 
Y. If black 32 does not equal black 200, then the pro- 
cess wilt step by one between black 200 plus 0.5 and 

25 black 3.2 minus 0.5 and the delta array is then set 
equal to Y x (N - black 200) / (black 32 - black 200). 

Continuing with this step of creating the delta 
an-ay, if black 200 does not equal 0 then the delta 
anray is zeroed between 0.5 and black 200 - 0.5. Next. 

30 the function f(x) is evaluated at white 32 and assigned 
a value Y. If white 32 is not equal to white 200, then 
between white 32 + 0.5 and white 200 - 0.5. the delta 
array is set to equal the value Y x (white 200 - N- 
) / (white 200 - white 32). If white 200 is not 63 then 

35 the delta anray is zeroed between white 200 plus 0.5 
and 62.5. 

The previous two paragraphs are stated algo- 
rithmically. Their effect is to place in the delta anray 
105 numbers corresponding to curve 104 of Fig. 9, 
40 complete with tapered ends 1 06 and 1 07, and 0 seg- 
ments 105. 

There are several rationales for this step of the 
process. The curve in Fig. 9 comprising segments 
104, 106, 107, and 105 is the gray scale graph that 

45 needs to be corrected. The step has removed both 
image artifacts we desire to preserve as well as uncor- 
rectable capture anomalies. Sections of the gray 
scale with high concentration of pels need to be 
expanded proportionally to that concentration. The 

50 delta, or difference array, holds the difference which 
must be obtained between the converted gray scale 
of two states and the final convert array to flatten the 
curve of the final converted image. Still further, at the 
ends between the 32 and the ultimate 200 points, a 

55 smooth descent is provided and wherein beyond the 
200 points, no gray scale is allocated. 

As a final step in the processing method, a cor- 
rection lookup table is generated based upon the 

8 
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overall best fit and utilised to correct the image. First 
a convert an-ay 109 in Fig. 10 is generated with ele- 
ments from 0 to 63. The first element of the convert 
an-ay is set = 0. Then, for N from 1 to 63, the method 
sets the convert array (N) = convert array (N - 1) + 5 
delta array (N - 0.5). Next, the process again scans for 
N = 0 to 63, and sets the convert array (N) = convert 
array (N) / convert array (63). This produces the nor- 
malised lookup convert array. Essentially, it inte- 
grates the delta array to give the biggest jump in gray io 
scale at those levels of gray that had the highest con- 
centration of pels. 

Up to this point, the convert array has been nor- 
malised between 0.0 to 1.0. Next, in accordance with 
the image processing of the invention, the convert is 
array is intergrzed. If colour values must be 6 bits 
each, each element of the convert array is translated 
using a function of the form truncate (convert array (N) 
X 63.99). Finally, the actual conversion from the origi- 
nal image to the corrected image that is the product 20 
of this invention is performed with this convert anray. 
For each colour of each pel of the original image, the 
6 bit value of the colour is utilised as a pointer into the 
convert array, and a new 6 bit value is received back 
to replace the old value. In this manner, pels corre- 25 
spending to a corrected image are thereby created 
which may be utilised in any desired manner thereaf- 
ter such as in the display of the corrected image on the 
monitors 5 or 7 in Fig. 1 . It will noted that the foregoing 
image processing of the invention may also be under- 30 
stood in terms of a more conventional approach of 
transfer function theory associated with negatives, for 
example in the photographic art, such a transfer func- 
tion 1 1 0 being shown in Fig. 1 1 , This function typically 
has several areas associated with it including an area 35 
112 resulting in underexposed areas, a low contrast 
toe 114, a high contrast middle 116, a low contrast 
shoulder 118 and a saturation level 120. 

It will be appreciated that this transfer function 
may be seen as the integral of the previously des- 40 
cribed histogram. In other words, integration of the f(x) 
yields a function of the form Z + Ax + Bx2 + Cx^. These 
coefficients Z. A, B, and C map to a black level, white 
level, gamma, and saturation, respectively. In other 
words the coefficient C of the integration maps to the 45 
C of the histogram function inasmuch as the integral 
of Cx2 = Cx3/3. Thus the foregoing should remove any 
confusion with respect to use of an term for gamma 
correction in the prior art. 

In the foregoing, it will be appreciated that the ex- so 
treme points 90 of Fig. 6 of the histogram have been 
limited, however the invention Is not intended to be 
limited in scope to such treatment of these points. As 
an example, these points may, rather than be limited, 
simply be decreased in weight as the best fit is per- ss 
formed which may yield an equivalent but slightly dif- 
ferent result. In other words, an iterative best fit may 
also be desirable in some applications. A first pass 



best fit would accordingly be performed. Then based 
upon how far these extreme points are from the first 
pass curve fit, a second pass fit would be performed 
wherein they are simply weighted less as a function 
of their distance from the curve fit by the first pass. It 
will be seen from the foregoing by simply limiting the 
extreme points this is merely an expedient in taking 
these extreme points into account which may be 
refined by such weighting of these points. 

A method and system has been described in 
which analog video data input is received by a video 
capture system and digitised. From the pels of the 
captured image, a gray scale level histogram is con- 
structed with the system processor, and normalised to 
established white-black references similar to the con- 
trast control function usually available at image cap- 
ture time. Peaks likely to be attributable to image 
artifacts are limited. The image processor then per- 
forms a second order best fit regression of the histog- 
ram from which the image is thereafter corrected. An 
automatic correction of the image contrast is thereby 
effected, wherein capture artifacts are reduced while 
retaining those of the image. In an alternate embodi- 
ment described, instead of limiting peaks, an iterative 
best fit is perfonned, wherein the peaks are weighted 
in a subsequent best fit pass as a function of their dis- 
tances from the curve fit from a prior best fit pass. 



Claims 

1. A method for automatic image enhancement of 
image data, comprising 

generating a histogram from video data 
representing said image 

generating a best fit regression function of 
said histogram; and 

modifying said video data as a function of 
said regression function. 

2. The method as claimed in Claim 1 wherein said 
best fit regression function is polynomial. 

3. The method as claimed in Claim 2 wherein said 
polynomial is a second order. 

4. The method as claimed in Claim 3 wherein said 
best fit is completed over a range less than the full 
gray scale range. 

5. The method as claimed in Claim 1 wherein said 
range is between limits selected to include a 
selected percentage of said histogram. 

6. The method as claimed in Claim 1 wherein said 
iterative best fit comprises 

generating a plurality of second order best 
fit functions of said histogram; and 
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13. The method as claimed in Claim 4 wherein said 
area comprising a selective percentage of said 
histogram encompasses a range from black to 
white in said image. 

14. The method as claimed in Claim 13 wherein said 
best fit curve outside said area of said polynomial 
fit decreases from polynomial values at the edges 
of said area to zero at the ends of said histogram. 

15. A system for automatic image enhancement 
incorporating the method as claimed in any one 
of the preceding claims. 
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wherein, for each said function, peaks are 
weighted as a function of their distances from cor- 
responding points on a previous one of said best 
fit functions. 

7. The method as claimed in Claim 1 wherein prior 
to generation of said regression function, peaks 
of said histogram are weighted as a function of 
their deviation from the norm. 

8. The method as claimed in Claim 7 wherein said 
nonmalising adjusts contrasts of said image data 
to set white and black reference levels. 



9. The method as claimed in Claim 1 wherein said is 
best fit is iterative. 

10. The method as claimed in Claim 9 wherein said 
iterative best fit comprises 

generating a plurality of second order best 20 
fit functions of said histogram; and 

wherein, for each said function, histogram 
values are weighted as a function of their dist- 
ances from corresponding points on a previous 
one of said best fit functions. 25 

11. The method as claimed in Claim 1 further includ- 
ing operating on a colour image in which said his- 
togram is a weighted average of histograms of 
each colour record individually. 30 

12. The method as claimed in Claim 11 wherein the 
weight assigned to the colour blue is less than the 
weight assigned to the colours red or green. 
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